Develop - #1345
Merged
Merged
Conversation
…e spurious AttributeError
# Modernize backend: Django 4.2 · Celery 5 · Python 3.11 · PostgreSQL 16 · TF 2.15 / CUDA 12 ## Summary Brings Rodan's backend off its end-of-life stack and onto current, H100-capable infrastructure. Upgrades the web/framework layer, the database, and the GPU job runtime, adapts the container/k8s deployment accordingly, and fixes a set of regressions surfaced by the upgrades. 18 commits, ~129 files. ## Motivation The old stack (Django 2.0, Celery 4.4, Python 3.7, PostgreSQL 9.6, and CUDA/TF builds targeting pre-Hopper GPUs) no longer installs cleanly, receives no security fixes, and can't run on the current H100 (sm_90) hardware. This branch is the coordinated upgrade of all of those at once. ## What's changed **Framework (Phase 1)** - Django 2.0 → **4.2.16**, DRF → **3.15.2**, djoser → 2.2, Celery 4.4 → **5.3.6**, Python 3.7 → **3.11**, numpy → 1.26.4. - Realtime layer migrated from `ws4redis` → **Django Channels 4** (ASGI `ProtocolTypeRouter` + a broadcast consumer relaying the redis pub/sub channel). - `gamera4` rebuilt for py3.11. **Database** - PostgreSQL 9.6 → **16** (required by Django 4.2). Includes a dump/restore cutover runbook (`k8s/POSTGRES_UPGRADE.md`); `postgres/Dockerfile` now `FROM postgres:16`. **GPU jobs (Phase 3)** - GPU job image rebuilt on **TensorFlow 2.15 / CUDA 12 / Python 3.11** for H100. - `text_alignment` ported to py3.11 / TF 2.15 (upstream calamari-ocr 2.3.1). **Containerization / k8s** - Image chain rebuilt (`rodan-python3-celery` → `rodan-main` → `nginx`; plus client, gpu-celery, postgres, iipsrv) and all k8s manifests updated; GPU pod gets MIG visibility and `compute,utility` capabilities. **CI/CD** - `build-and-deploy` and `build-ci-jobs` workflows; CI test fixes. **Bug fixes surfaced by the upgrade** - **List filtering (`filter`→`filterset`)** — django-filter 24.3 ignores the legacy `filter_fields`/`filter_class`; renamed to `filterset_fields`/`filterset_class` across all list views (restores `?project=` and other filters), fixed 3 filter-method lambda signatures, and **restored object-level list scoping** via a guardian-based `ObjectPermissionsFilter` in `DEFAULT_FILTER_BACKENDS` (a list with no `?project=` no longer leaks every project's rows). - **Interactive editor open** — per-key template cache fix so a second editor open per worker doesn't 500. - **Job import guard**, **ASGI startup**, `is`→`==` identity check, gpu-celery CUDA env. ## Breaking changes / migration notes - **Requires the PostgreSQL 9.6→16 dump/restore** before deploy (see `k8s/POSTGRES_UPGRADE.md`). - **All images must be rebuilt** — base-image chain, Python, and CUDA/TF versions all moved. - Deploy is triggered by a `v*` tag or a manual `build-and-deploy` dispatch (merging to `develop` only builds `nightly`/`sha-` images). ## Testing & validation - CI test suite green on the branch. - All automated job types validated against real DB rows + PVC files via a replay harness; both interactive editors (Neon, Pixel.js) verified end-to-end in a real browser on the modernized stack. - Live distributed end-to-end workflow run passed on the k3s cluster.
|
SHA: 86be539
details |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.